Method of graphically tagging and recalling identified structures under visualization for robotic surgery

ABSTRACT

A system and method for augmenting an endoscopic display during a medical procedure including capturing a real-time image of a working space within a body cavity during a medical procedure. A feature of interest in the image is identified to the system using a user input handle of a surgical robotic system, and a graphical tag is displayed on the image marking the feature.

This application is a continuation of U.S. application Ser. No.16/018,037, filed Jun. 25, 2018 which claims the benefit of ProvisionalNo. U.S. Provisional 62/524,133, filed Jun. 23, 2017, U.S. Provisional62/524,143, filed Jun. 23, 2017, and U.S. Provisional 62/524,154, filedJun. 23, 2017, each of which is incorporated herein by reference.

BACKGROUND

There are different types of robotic systems on the market or underdevelopment. Some surgical robotic systems, such as those described inU.S. Pat. Nos. 8,506,555, 9,358,682, and 9,707,684 use a plurality ofrobotic arms. Each arm carries a surgical instrument, or the endoscopiccamera used to capture images from within the body for display on amonitor. Other surgical robotic systems use a single arm that carries aplurality of instruments and a camera that extend into the body via asingle incision. See WO 2016/057989. Each of these types of roboticsystems uses motors to position and/or orient the camera and instrumentsand to, where applicable, actuate the instruments. Typicalconfigurations allow two or three instruments and the camera to besupported and manipulated by the system.

The image captured by the camera is shown on a display at the surgeonconsole. The console may be located patient-side, within the sterilefield, or outside of the sterile field.

As with manual laparoscopic surgery, surgical instruments and camerasused for robotic procedures may be passed into the body cavity viatrocars. Input to the system is generated based on input from a surgeonpositioned at the console, typically using input devices such as inputhandles and a foot pedal. US Published Application 2013/0030571describes the use of an eye tracking system to give input to the system.The input is used to control motion of the camera-holding arm, allowingrepositioning of the camera (e.g. to pan and/or zoom the image seen bythe user on the image display) based on the where the user is looking onthe camera display and/or how close the user's eyes are to the display.

Motion and actuation of the surgical instruments and the camera iscontrolled based on the user input. Some robotic systems are configuredto deliver haptic feedback to the surgeon at the controls, such as bycausing the surgeon to feel resistance at the input handles that isproportional to the forces experienced by the instruments moved withinthe body.

In a robotic surgical system utilizing endoscopic visualization, itwould be advantageous for the surgeon to dynamically tag or “bookmark”single points or identified anatomical structures in the surgical spacefor the purposes of recalling and monitoring their positions visually ata later time. This would allow the surgeon to better navigate thesurgical space under compromised visual conditions (caused by blood,smoke, or other types of obstructions) by locating or possibly avoidingcritical anatomical structures. This data may be used to create a “worldmodel” for a surgical robotic system and be used to identify structuresin the abdomen that are to be avoided by robotically controlledinstruments. Such a system, which may include configurations thatanticipate the possibility of instrument contact with such structures aswell as those that cause the robotic system to automatically avoid thestructures, is described in co-pending U.S. application Ser. No.16/010,388, which is incorporated herein by reference.

Various surface mapping methods exist that allow the topography of asurface to be determined and can be implemented for use in surgicalapplications. Some such methods use stereoscopic information from a 3Dendoscope, structured light measured by a 3D endoscope, structured lightmeasured by a 2D endoscope, or a combination thereof. One type ofsurface mapping method is one using structured light. Structured lighttechniques are used in a variety of contexts to generatethree-dimensional (3D) maps or models of surfaces. These techniquesinclude projecting a pattern of structured light (e.g. a grid or aseries of stripes) onto an object or surface. One or more camerascapture an image of the projected pattern. From the captured images thesystem can determine the distance between the camera and the surface atvarious points, allowing the topography/shape of the surface to bedetermined. Other types of surface mapping methods also exist and may beused in conjunction with the system and method described in thisapplication. These types of techniques may be implemented in the systemsand processes described below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates information flow for the methods oftagging structures and providing overlays.

FIGS. 2-6 show endoscopic views of a surgical field illustrating use ofthe features described in this application, in which:

FIG. 2 shows the original endoscopic view at the start of a procedure;

FIG. 3 shows the view of FIG. 2 , with identified structureshighlighted;

FIG. 4 shows the view of FIG. 2 , but with an obstruction in the imagingfield partially obscuring the endoscopic view and with partialhighlighting of the identified structures;

FIG. 5 shows the view of FIG. 4 , but in which the amount of obstructionhas increased;

FIG. 6 shows the view of FIG. 5 in which the amount of obstruction hasincreased to nearly full obstruction, and in which a structure to avoidis highlighted as the visual field becomes more obscured.

DETAILED DESCRIPTION

A system in accordance with the present invention includes one or moreinformation sources, such as 2D, 3D and/or structured light imagingsources, and a visual display for displaying visual information fromthose sources to a user. A surgeon or other user uses an input device toidentify to the system the structures, areas etc. that are to be tagged.The associated processor registers the location of the tagged structure,area etc. to a model of the surgical site within the patient and isprogrammed to generate overlays displayed on the visual display thatidentify or mark the tagged structures or areas.

FIG. 1 schematically illustrates information flow for the methods oftagging structures and providing overlays. Note that not all informationsources listed are necessary for use of the invention, and additionalinformation sources may be used without departing from the scope of theinvention. At a high level, this application describes a system andmethod for visually tracking locations or structures within a surgicalendoscopic visual field of view as that view changes. The systemacquires data from one or more sources and builds an internaltopographical model of the surgical space which allows the surgeon totag, mark, flag, or bookmark structures or locations in the visual fieldutilizing that model. Some implementations allow the surgeon to recallpreviously tagged structures or locations via the robotic userinterface, and overlay a graphical tag over the selected location in theendoscopic view to facilitate navigation of the surgical space undercompromised visual conditions.

In this application, the term “tag” is used to refer to a tag, bookmark(an alternative term is “waypoint), and may be any of the following:

-   -   Point, pose    -   2-dimensional surface, 3-dimensional surface    -   Region    -   Edge

The 3-dimensional data defining the tissue topography and the datacorresponding to the position/location of the tags (taggedlocations/identified structures) may be gathered from stereoscopicinformation from a 3D endoscope, structured light measured by a 3Dendoscope, structured light measured by a 2D endoscope, or a combinationthereof. This 3-dimensional data may be acquired as described inco-pending application U.S. Ser. No. 16/018,039, filed Jun. 25, 2018,entitled Method and Apparatus for Providing Procedural Information UsingSurface Mapping. The 3-dimensional data may be acquired once during aprocedure, updated intermittently at a certain interval, updatedintermittently on-demand, or updated continuously. The tag position datamay be registered to the 3-dimensional model of the surgical anatomycomprising the system's world view. The tag and/or 3-dimensional modelmay be registered to the real-time endoscopic view of the surgicalfield.

Turning now to a discussion of the type of user input that may be given,structures or locations can be tagged in the field of view using a pointand click model of selection. The user may move a graphical pointer to atarget location and give input to the system to cause the system to tagthe location. The user input device might be any type of device known tothose skilled in the art that will allow a user to select a location orarea on a screen and identify that area to the system. As one example,an eye tracking system, manual keypad control, motion of a user inputhandle of the surgeon console, or other input device may be used tomanipulate a mouse pointer moveable on the visual display to the regionthe user wishes to tag. Input of the user's selection may be given usingphysical input (e.g. button, foot pedal), voice command, etc. As onespecific example, a button on the surgeon handle can be pressed tofinalize the selections. As another example, the endoscopic display maybe a touch screen that allows the user to create a tag by directlytouching the feature of interest on the touch screen. The selectedlocations in the visual field are correlated and linked to thecorresponding areas of the dynamic anatomical model of the surgicalspace, and can be saved and recalled by the surgeon, regardless ofchanges to the view or the anatomical model. Some implementations allowfor automatic recognition of anatomical structures selected by thesurgeon using algorithms (including, without limitation, computervision) and historical data from a variety of sources. Someimplementations allow for dynamic adjustments by the surgeon to thetranslucence of the graphical tags based on desired visual effect.

In alternative implementations, the robotic surgical system may usekinematic knowledge from the surgical robotic system to determine thelocation of a tag. In such embodiments, the user might steer theinstrument tip to the location to be tagged and give user input to thesystem instructing the system to record the data corresponding to thepose of the instrument tip (position and orientation in Cartesian space)in order to create the tag.

User input, such as in forms described here for setting tags, may alsobe used to instruct the system to remove selected tags, or to take someother action with respect to a selected tag. The system might also beequipped to allow a second user such as a surgical assistant, toadd/remove tags, classify/categorize items in the model, etc.

Tags are visually represented on the endoscopic display using graphicaloverlays to mark points, boundaries, regions, sections of structuressuch as blood vessels, etc. This can give the surgeon the continuedability to navigate surgical instruments towards, away from or aroundidentified features even when the surgical field is obscured by smoke,pooling blood etc. Visual characteristics of the tags may be altered asa means of visual feedback to the surgeon relating to some changedcondition, such as a change in visibility in the endoscopic field thatcompromises the visibility of identified structures on the endoscopicdisplay. Thus, for example, the opacity of an overlay may be changed,such as by a change in its opacity or color as the identified structurebecomes less visible or changes. As a more specific example, the opacityof the overlay might be increased as the identified structure becomesless visible on the endoscopic image. A decrease in the visibility of anidentified structure might additionally or alternatively result in avisual, auditory or haptic notification signal to the user.

The system may be set up to recognize that a tagged/bookmarked structureis visually obscured and to provide/alter the visual feedback to thesurgeon as described in the preceding paragraph. The system may thus beable to detect changes in the visibility of bookmarked features on theendoscopic view (or increases in the degree to which the bookmarkedfeatures are obscured). This may be carried out using, edge detectiontechniques (e.g. detecting edges of a forming blood pool), by detectingchanges in image contrast (e.g. local contrast or color or overallcontrast or color), or by detecting changes in the image texture.Detected changes in measured depth and/or local surface contours mightalso be used.

Overlays can be provided to the display by an external processor as anintegrated signal, or by a display that integrates multiple signals—suchas from an endoscope and a graphics generator—into a single display tothe user.

The robotic surgical system may be responsive to the tags in a varietyof ways. The position of a tag may be used as a reference with respectto which one or more of the robotic arms is repositioned and/orreoriented in order to reposition/reorient a surgical instrument orcamera. The surgeon input controls might include an input deviceallowing the surgeon to instruct the robotic system to automaticallymove an attached instrument to a given location and/or orientation thatis defined relative to the location of a tag (rather than requiring thesurgeon to navigate the surgical instrument to/towards the tag). This isparticularly beneficial in instances where some of the endoscopic viewis obscured by blood, smoke etc. Thus the instrument tip may beautomatically moved to the location of a tag, or to a predeterminedstandoff distance (optionally in a predetermined standoff direction) inresponse to input to the robotic surgical system to robotically positionthe instrument tip as instructed. The position of a tag may be similarlyused in a step for automatically repositioning and/or reorienting theendoscope.

As another example, the location of a tag may be used as the basis forcontrolling the maximum insertion depth of at least one instrumentcarried by a robotic arm. In other words, the location datacorresponding to the tag position is used to establish a plane beyondwhich the instrument tip should not pass. Should the user attempt tonavigate the instrument tip beyond that plane, one or more of a numberof events might take place. For example, the robotic arm may stop movingso it does not pass the plane, the user might be provided with visual orauditory feedback that the instrument tip has reached the plane, theuser might be provided with haptic feedback in the form of vibratoryfeedback or force feedback giving the user the sensation of pushing theinstrument tip against a barrier. Similar forms of feedback mightadditionally or alternatively be given as the system detects movement ofthe instrument within a certain distance of the plane, with themagnitude of the feedback (volume or pitch of auditory feedback,intensity or frequency of vibratory feedback, brightness, size or otherappearance feature of visual feedback) increasing as the instrument tipmoves closer to the plane.

The system may be set up to provide the user with menus that allowcategorization of tags into groups such as “structures to avoid”,“targeted structures”, “paths to follow”, etc. Concepts described heremight be implemented in an augmented reality configuration with overlaysdisplayed on a headset or transparent head mounted device.

Co-pending U.S. application Ser. No. 16/010,388 filed Jun. 15, 2018,describes creation, and use of a “world model”, or a spatial layout ofthe environment within the body cavity, which includes the relevantanatomy and tissues/structures within the body cavity that are to beavoided by surgical instruments during a robotic surgery. The systemsand methods described in this application may provide 3D data for theworld model or associated kinematic models in that (see for example FIG.5 of that application) type of system and process. For example, theinputs, outputs, and outcomes referenced in the co-pending applicationmay be used in concert with the bookmarking techniques described in thiscurrent application. In addition, the automatic or assisted detection ofobstructions (e.g. pooled blood or an expanding region of pooling blood)or other anatomical structures/features described here may beincorporated into the world model and/or into scans generated inco-pending U.S. application Ser. No. 16/018,042 entitled Method andApparatus for Providing Improved Peri-operative Scans and Recall of ScanData, filed Jun. 25, 2018, which is incorporated herein by reference.

It also should be noted that 3-dimensional data acquired as described inco-pending application U.S. Ser. No. 16/018,039, filed Jun. 25, 2018,entitled Method and Apparatus for Providing Procedural Information UsingSurface Mapping and data acquired in co-pending U.S. application Ser.No. 16/018,042 entitled Method and Apparatus for Providing ImprovedPeri-operative Scans and Recall of Scan Data, filed Jun. 25, 2018 may beused in conjunction with, or without, the 3D data described using thesystem disclosed herein, to provide 3D data for the world model orassociated kinematic models in system and process described inco-pending U.S. application Ser. No. 16/010,388 filed Jun. 15, 2018.

Usage Example:

During a surgical procedure, the surgeon identifies an area of theanatomy of critical importance such as a major artery, or a ureter, orother feature. The surgeon then engages the robotic user interface toenter location selection mode. Once in selection mode, the surgeonpoints to the feature of interest on the endoscopic display. Thispointing step may be carried out by navigating a graphical pointer tothe feature of interest using input from a user input device, which inthis example is an eye tracker. Using eye tracking technology, the usercan perform this step by moving his/her eyes to the region of interestand then instructing the system using another input mechanism such as abutton or foot pedal to mark the region of interest. More particularly,using the example of the eye tracking technology, the surgeon navigatesthe graphical mouse pointer to the feature of interest, places the mousepointer over the endoscopic view of the major artery and “selects” itusing a button on the robotic handle interface or other input devices.Location selection mode is exited and the graphical mouse pointer isremoved from the view. For the remainder of the surgery, a graphical“tag” is unobtrusively visible on top of the major artery identified bythe surgeon in the endoscopic view, regardless of changes in the view orobstructions to the view such as blood or smoke. In the event that themajor artery is obscured by another anatomical structure, the tagchanges its visual representation (translucence, color, etc.) toindicate that the identified structure is no longer in the foreground(“buried”). A mechanism is also provided to allow the surgeon to removeand restore the graphical tag from view at any time during theprocedure.

Another usage example shown in the series of figures attached as FIG.2-6 . FIG. 2 shows the initially clear endoscopic view. In FIG. 3 ,structures or regions A, B and C have been marked with color overlays.Here the overlay is shown as fully opaque, but under normal operation,would be transparent or nearly so, and/or it might display onlyperimeters of the identified structures. This figure shows both varyingicons and highlighted regions to denote different structuretypes/categories or regions, but either method may be used and is withinthe scope of the invention. In this example, region A may signify anarea for operation into which the surgeon may want to enter. Thehighlighted perimeter in the visually-obscured space may allow thesurgeon to be aware of its boundaries, either to enter it with increasedassurance of its location, or to avoid pushing too hard against theperimeter of it. Regions B and C may be vessels that the surgeon wishesto avoid with the surgical instruments. In FIG. 4 , blood has begun tofill the surgical field, and the initially clear endoscopic view hasbecome partially obscured by blood. The perimeters marking the regionsA, B and C, as well as the shape remain displayed. Note that this viewalso shows the partial highlighting of the identified structures. FIGS.5 and 6 are endoscopic views so progressively increasing levels ofocclusion of the view by the blood pool, with FIG. 6 showing almost fullocclusion of some identified structures and highlighting of theidentified structures. Note that a structure to avoid is highlighted, orits highlighting is further enhanced, as the visual field becomes moreobscured to assist the surgeon in avoiding it while attempting to stopthe blood loss. As the field fills with blood, the highlight around theblood vessels marked by regions B and C may allow the surgeon to find itin the obscured field and provide clamping pressure/cauterization/etc.to stop the bleeding.

All patents and applications referred to herein are incorporated hereinby reference.

We claim:
 1. A method of tagging regions of interest on displayed imagesduring a medical procedure, comprising: positioning an endoscope in abody cavity; positioning a surgical instrument in the body cavity;mounting the surgical instrument to a first robotic manipulator arm;causing the first robotic manipulator arm to manipulate the surgicalinstrument within the body cavity in in response to user manipulation ofa user input handle; capturing images of a surgical site within the bodycavity and displaying the images on a display, the displayed imagesincluding images of the surgical instrument, entering a tag selectionmode comprising: displaying a graphical pointer as an overlay on theimages displayed on the display, the graphical pointer in the tagselection mode moveable on the display in response to user manipulationof the user input handle; in response to user manipulation of the userinput handle, positioning the graphical pointer at a region of intereston images of the surgical site displayed on the display; in response touser selection input, displaying a graphical tag at the location of thegraphical pointer on the images displayed on the display.
 2. The methodof claim 1, wherein the method further comprises: exiting the tagselection mode, wherein graphical tag remains displayed at the region ofinterest on images of the surgical site displayed on the display.
 3. Themethod of claim 1, wherein the method further includes removing thegraphical tag in response to user input to remove the tag.
 4. The methodof claim 3, wherein the method further includes restoring the graphicaltag at the region of interest in response to user input to restore thetag.
 5. The method of claim 1, wherein the method includes receiving theselection input from at least one of a microphone, a button, food pedal,joystick, stylus, touch input, or eye tracker.
 6. The method of claim 1,further comprising positioning a second surgical instrument in a bodycavity; mounting the second surgical instrument to a second roboticmanipulator arm; and causing at least one of the following: causing thefirst robotic manipulator to move a tip of the first surgical instrumentto a location a predetermined standoff distance from the tagged regionof interest, or causing the second robotic manipulator to move a tip ofthe second surgical instrument to a location a predetermined standoffdistance from the tagged region of interest.
 7. The method of claim 1,further comprising mounting the endoscope to the second roboticmanipulator arm; and causing the second robotic manipulator to move theendoscope towards the tagged region of interest.
 8. The method of claim1, wherein the region of interest identified by the tag has a depth, andwherein the method further comprising: positioning a second surgicalinstrument in a body cavity; mounting the second surgical instrument toa second robotic manipulator arm; and preventing movement of at leastone of the first or second instruments to a location in the body cavitythat is deeper than the depth.
 9. The method of claim 1, furthercomprising detecting in real-time a change in the image during themedical procedure, the change indicative of the feature of interestbecoming at least partially obscured by smoke or pooling blood; andaltering a quality of the displayed graphical tag to enhance itsvisibility in response to the detected change.